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ABSTRACT 

A three-part study was conducted to present an 
iridepL look at reading comprehension instruction in the 'middle 
gradefj. In the first part of the study, four basal series for grades 
4,\5, and 6, were analyzed for clarity of communication, adequacy of 
skill practice provided, and a number of other comprehension related 
dimensions. In the second part, 17 teachers were videotaped as they 
taught two comprehension topic areas and were interviewed to get 
their perceptions of the texts they used and of their students* 
mastery of the material taught. These results were compared with 
those obtained from a larger Sample of teachers who completed 
questionnaires. The third part of the study examined student 
achievement for the observed teachers on criterion-referenced tests 
designed to assess what was taught. Overall results indicated that 
(1) the text presentations were inadequate in terms of their 
instructional design features, (2) the teachers did not improve upon 
the texts, (3) teacher perceptions of how well they taught and how 
much their students learned were inaccurate, and (4) only 55% of the 
students learned 50% of the comprehension skills presented. (FL) 
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Abstract 

1 i report presents findings from a three-pronged study of reading 
comprehension instruction in the middle grades* In part one^ areas of 
reading comprehension were selected (su^ch as main id ea ) and text material 
for their introduction, instruction, and practice were analyzed in depth 
for frequency, clarity, and adequacy* In part two, teachers were 
videotaped teaching two comprehension topic areas and itijterviewed about 
their instruction. In part three, the students were tested after the 
reading instruction to see how thoroughly they had mastered what the 
teachers had taught* The results showed that the text presentations, were 
inadequate in terms of their instructional design features, that the 
teachers did not improve on the texts, that teacher perceptions of how well 
they taught and how much their students learned were inaccurate, and that 
only 55 percent of the students learned 50 percent of the comprehension 
skills presented* 
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Despite the well-'documented fact that students spend approximately 70% 
of their time In school working with textbooks and workbooks (Gall, 1981) 
there have been few studies to examine Instructional materials for their 
design characterls.flcs, the adequacy of teacher presentations of the 
materials, and the student outcomes they achieve. Beck (1978) and Beck and 
McCaslln (1978) did some of the tlrst work In this area when they analyzed 
eight beginning reading programs. In their analyses, they looked at 
dimensions that affect the development of code-breaking skills for 
compensatory education students, those students likely to have trouble 
learning to read, Beck and McCaslln (1978) analyzed; (1) how letter-sound 
correspondences are taught and (2) the pedagogy of the programs. They 
concluded: **If the pedagogy Is so convoluted that the correspondences 
cannot be learned, or if It requires skills that young readers do not nave. 
It matters little that the program contains exquisite sequencing" (p. 68). 
A further conclusion from Beck and McCaslln (1978) Is that compensatory 
education students will have trouble learning to read If they are In the 
basal programs. For instruction In code-breaking skills, they ^vored the 
code-emphasis programs Dlstar (Engelmann & Bruner, 1974, 1975) and Palp 
Alto (Glim, 1973). * , 

But, decoding Is just one aspect of the reading process, and although 
there is recent research to suggest long-term success in reading 
comprehension for students who first learned to read with a synthetic 
phonics program (^Jeyer, in press), there is still substantial disagreement 
about the long-term effects of code-emphasis programs (Anderson, Mason, & 
Shlrey, Note 1; (Carniue, Carnine, & Gersten, Note 2). Despite this 
controversy over long-term effects, and the unresolved conflicts between 
the meaning-emphasis and code-emphasis proponents, there is unquestionable 
consensus among reading researchers and curriculum developers that reading 



comprehension is the major desired outcome for all students. Therefore, it 
is important to analyze beginning reading ^programs for instrutional 
dimensions that may affect reading comprehension. 

Beck, McKeown, McCasliu, and Burkes (1979) analyzed two commercial 
reading programs to apply, **theory, research, logical a::gument, and our own 
teaching experiences and intuitions to an examination of Instructional 
materials" (p# 2). Their, analysis focused upon what they defined as: (1) 
textual problems, (2) picture characteristics, (3) previous knowledge 
assumed by the text, (4) vocabulary knowledge and application, (5) 
direct ions' for setting the purport for reading, (6) how the reading lesson 
is divided, and (7) questions tdf follow story-reading. Their conclusions 
from the analysis of the Houghton Mifflin Reading Series (Durr, 1976) and 
the Glnn Reading 720 Program (Clymer, 1976) were: (1) that basal reader 
vocabularies will be problematical for compensatory education students; (2) 
that the pictures used to Illustrate the texts should be more carefully 
designed to depict meaningful events; (3) that too much background 
knowledge was assumed in the later primary and intermediate grade stories; 
and (4) that there was too much dependence on context as the primary, source 
of vocabulary development. They were also concerned with the way basal 
stories are divided into reading uhits, the questioning techniques 
presented in the teacher's guides, and the need for students to develop an 
overall sense of the story's theme before focusing on direct questions (p. 
?)• 

Two more recent studies paint similarly critical pictures of basal 
reading raaterial.i. Durkin (1981) analyzed the teacher's manuals for five 
basal readers for kindergarten through sixth grade. When she matched six 
definitions of comprehension instruction to the materials, she found that 



the number of recommended instructional procedures ranged from 60 In one 
series to 128 in another. Number of review procedures ranged from 85 to 
418, application from 111 to 538, number of practice from 495 to 832, 
number of preparation from 328 to 491, and number of assessment froiu 328 to 
437. Durkin (1981) further described the, •'scanty direct, explicit 
instruction'* (p* 542) and the "tendency to ofifer numerous application and 
practice exercises" (p» 542)# 

A fifth study by Osborn (Note 3) looked exclusively at one component 
of a basal reader, the student workbook* In her work, Osborn (Note 3) 

stated, "The primary questions are about sufficiency, efficiency, and 

If 

effectiveness" .(p# 16)» She then developed twenty guidelines for workbook 
writers and reviewers* These guidelines range from, "A sufficient 
proportion of workbook tasks should be relevant to the instruction, that is 
going on in the rest of the unit or lesson" (p» 18) to, "Instructions to 
students should be clear, unambiguous, and easy to follow; brevity is a 
virtue" (p# 35) and, "When appropriate, workbook tasks should be 
accompanied by brief explanations of purpose for both teachers and 



students" (p» 92). Osborn concluded that application ta^s — those kinds of 
items that give students practice in workbooks on concepts covered in 
readers seldom appeared, if at all, in the workbooks she reviewed. 

To sum up, then there have been five major studies of basal reading 
materials in the last 'few years. These studies haye looked at basal 
reading programs to determine the type and amount of practice given for 
decoding and comprehension in readers and workbooks from kindergarten 
through the intermediate grades* All of these studies have concluded' that 
the basals lack sufficient direct instruction, sequential practice, and 
application practice in reading comprehension. 



This presents study is reported in t^iiree parts* The first part 
extended the analyses of Beck (1978), Beck and McCaalin (1979), Beck et al. 
(1979), Durkin U981), and Osborn (Note 3) by reviewing four basal reading 
programs for grades 4, 5, and 6 for clarity of the comnjunication, adequacy 
of skill practice provided in the texts, and several other dimensions* A 
second part of this study departed from previous work and observed 
teachers using these basal readers* The purpose of the observational 
study was to assess how teachers presented the materials and interacted 
with students* The observation study also included interviews with the 
teache^rs to get their perceptions of the texts they used and of their 
students mastt>ry of the material taught* To establish the 
representativeness of the 17 teachers observed, they were compared to a 
larger sample of teachers using a questionnaire that was mailed to 3,000 
randomly selected fourth, fifth and sixth grade teachers. A comparison of 
8 of the 17 the observed teachers* responses with the responses from the 
493 teachers who. returned the questionnaire completed part two. A third 
part of this study looked at student achievement for the 17 observed 
teachers on criterion~re'f erenced measures designed to assess what was 
taught* 

Taken together, then, the three parts of this »tudy were intended to 
present an indepth look at reading comprehension instruction in the middle 
grades* We analyzed the basal materials, observed teachers teaching 
reading, interviewed all teachers observed to learn their attitudes and 
perceptions about their work, and tested their students to assess overall 
student performance on reading comprehension tasks. 

To simplify the presentation, each part of the study will first be 
reported separately and then they will be integrated* 



Part 1 4 Analysi s of^ Four Basal Reading Programs 
We ^'selected two coraaionly taught reading comprehension skills, finding 
the Main Idea, and working from Context Clues for tracking through grades 
A, 5, and 6 in four basal reading programs. To perform this avialysls, we 
counted every activity In each level of each basal* We then analyzed each 
appearance of these skills ^or clarity of presentation* 

Method . . 

Frequency * The first instru tional-deslgn variable examined for Main 

Idea and Context Clues was frequency of practice* This analysis involved 

all lessons on Main Idea in Glnn (Clymer, 1976)^ Scott Foresman (Scott 

Foresman & Co., 1974), Houghton Mlfxxxn (Durr, 1976), and Holt, Rinehart, 

. ari4, Winston (1977)* Main Idea and Context Clues were targeted for 
>•■•• ■ ^ *' 

analysis because of the general interest in these areas by researchers 

studying reading; comprehension, and the assumed ease with which these 

skills could be defined and contained* The first step was to Impose a 

structure on the basals to make it possible to determine the frequency at 

which teaching presentations and student ex^clses appeared in the texts. 

By counting the total number of pages in texts for grades 4, 5, and 6, and 

then dividing this number by the 480 (assuming 160 instructional days per 

year for the three years ^ it was possible to divide the texts into "pagfes 

per lesson***.^ A lesson was then defined as the pages covered during an 

average school day^s reading period. This organization permitted a count 

of lessons between instruction or practice on Main Idea. 

Clarity . The second, and admittedly much more subjective step in the 

analysis Main Idea and Context Clue segments was to assess the programs* 

clarity of presentation. This step Involved judging characteristics of the 

exercises such as ambiguity of the questions, examples, or rules; incorrect 

6 
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or misleading answers to questions; consistent wording in the teacher 
presentation and workbook; and the wording used in reA^ew^items for Main 
Idea and Context Clues. ^ . . 

»» 

Results ^ 
The results of the analysis for frequency and clarity of Main Idea 
presentations appear in Table 1. The average number of Main Idea 
presentations for the four programs for grades 4, 5, and 6 was 66. Of 
these 66 presentations, only 9 appear on the same day in both the student 
workbook and teacher *s guide. Only 22 lessons out of 480 lesson days deal 
with Main Idea. 



Insert Table 1 about here. 



The analyses for clarity showed that a high percentage of the 
questions were "ambiguous and not taught" (88%). An example was considered 
not taught if one like it had not been presented in 50 days. Twelve 

percent of the answers were misleading and wrong. The probability of a 

0 . 
correct interpretation from the examples given was only .27mi«e., there 

was an average of four possible interpretations per sequence). In 

addityon, there was a great deal of variation in how the examples appeared 

in the student workbook. Forty four percent of the time wording varied 

from one example to another. Eighteen percent of the time' 'student workbook 

itejtls varied in form from item to item. Teacher presentation wording in 

tpe teacher's guide varied 14% of the time, and teacher items forms varied 

10% of the time. Furthermore, the student workbooks had a range of 

response variations and visual distractions. Last, but not least, there 



were no correction procedures specified for any of the presentations in ^ 
either the teacher's guide or for the student materials* 

Durkin's work (1978-79, 1981) yielded results very similar to these. 

Part 2. Teacher Qbservations^ Interviews^ and Questionnaires 
This section reports^ the metl^odology and results for the videotaped 
observaHous made o| 17 fourth^ fijpth, and sixth grade teachers while 
teaching selected comprehension ta^ks and the information gathered from 
interviews and a questionnaire witl:| these teachers. 

Method 

Observations . Seventeen fourth, fifth, and sixth grade teachers from 
several school districts participated in this study. All of these "teachers 
were volunteers. All teachers were using their regular basal reading 
program. First, they were taped teaching a comprehension topic scheduled 
for that day from their basal reading program. Next, they were te^ed 

o 

teaching a designated Main Idea or other targeted topic (see Table 2 for a 
topic listing). All taping was done during a four month, period. Both , 
tapes of each teacher were first analyzed for teaching techniques , such as: 
the rate of their teaching presentation (the number of times the student 
respd^nded per^^^nute), the .proportion of turns presented to the whole group 
and to individual students > the numbei of responses on which the teacher 
respon^ied for or with the students, the amount and type of teacher praise, 
and the percentage of correct student responses. 

The video taped lessons were next analyzed to compare the teachers * 
performances with the information/directions in the teacher*s edition of 
their materials to see how closely the teachers followed the guides. 
Inter-rater percent agreements for coding these tapes was 85 pei'cent for 
teaching techniques, and 80 percent for the comparisons to teacher guides. 

8 
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Interviews . All teachers /ere interviewed before and after the two 
vi^ieo taplngs. These interviews were audiotaped and focused upon the 
teachers^ perceptions of student performance, their knowledge of what they 

were teaching, their judgioents of its adequacy, teaching procedures, etc* 

o 

The iiitarviews lagged approximately 40 minutes each. Only a segment of 
these interviews focusing on student performance expectations and program 
adequacy are reported below. 

Questionnaires 

X Teaching behaviors . To determine how representative the 17 observe4 

teachers were, questionnaires were sei>t to 3,000 fourth, fifth, and sixth 

• ■ ' it 

grade teachers randomly selected from al\ over the United States. Sixteen 
percent of the questionnaires were returned. The questionnaires sampled 
the range of questions asked of the observed teachers. Tlie same 
•questionnaires were sent to the 17 teachers studied in detail. Because 
this 'occurred late in the school year, anly 8 were returned. 

Result s 

Observations ^ 
1 ^ General teaching behaviors .' The video tapes were first coded for 
general teaching behaviors (Table 3). These 17 teact^ers presented 16% of 
their questioris to th group and 84% to individuals. Their presentations, 
yielded 4.4 student responses per minute. Students answered twenty-seven 
percent of the questions incorrectly. Teachers 'corrected 37%^ of the 
students* errors. Of the 37% of the errors corrected, 10% of the time 
teachers asked their original questions to students again, presumably to 
see if the students could answer the questions after the correction. 
Teachers performed the tasks for the students 20% of the time (model), and 
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performed the task with the students 14% of the time (lead). Students 
responded ind^lfTendently the remaining 66% oi the time (test). The teachers 
gave general praise 44% of the time,, and more appropriate behavior-specific 
praise only 2% of the time. They gave negative feedback after 1% of the 
responses, and gave no feedback at all to over half the responses (53% of 
the time). \ ^ 



Insert Table 3 about here* 



Do teachers follow the programs guidelines? . A second set of ratiligs 
served to compare the teachers* presentations to the'r program's 
specifications. Table 4 shows these results. This table shows the average 
number of questions per lesson that appear in the programs (A) and the 
average number of questions that the teachers asked (20). There is an 
silmost equal percentage of program questions (42%) arid teacher questions 
(48%) that are either ambiguous or misleading. Over two-thirds (69%) of 
the program questions were relevant to the topic presented in the reading 
selection, but only 24% of the teacher questions were relevant to the 
topic. The students were equally strong (78% to 77%) in their responses to 
^program questions and teacher questions. The teacher prompted correct 
respondency 22% of the time, while the program questions prompted the 
answer only 12? of the time. The^e comparisons demonstrate that teachers 
more than doubled* the number of questions they asked in comparison to the 
number specified in the programs. Equally surprising is that a much higher 
percentage (69%) of the programs questions were judged relevant to the 
reading* topic than the teachers* q|^stions. Most of the questions added by 
the teachers were irrelevant. The interview data, which are reported next 
point out further gulfs between teachers knd their programs. 

10 
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Insert Table 4 about here. 

Interviews . I 

During the interviews, the teachers answered questions related to how ' 
they expected their students to perform and their perceptions of the 
reading materials they used. Table 5 presents some of the interview 
outcomes. The interviews revealed that the teachers had high expectations 
for their students. They felt that 86% of their students should master all 
skills, and almost three-fourths (72%) of their students should be able to 
complete the workbook exercises for the lessons taped. The teachers 
ant icipated , "^f hough, that over half (58%) of their students needed practice 
on the fopic taught (main idea) and that a week's practice would be 
adequate. The teachers expected 56% of their students to master main idea. 
Yet they thought that their program was only 16% def icient . f or teaching 
students 'main-ldeaw - — - 



Insert Table 5 about here. 

\ ' - 
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Questionnaire \ 

The questionnaire data which follow are from 493 fourth, fifth, and 
sixth grade teachers. These responses are compared with those of 8 of the 
observed teachers who completed the same questionnaire. 

The two groups of teachers, those observed in the study, and those who 
responded to the questionnaire were demographlcally similar, and they 
structured their classrooms in quite similar ways. Both groups taught 



reading to about 27 students , and described their reading classes as 
"average." They taught reading In small groups, with between 5 1/2 and 6 
hours of reading Instruction pdr week. Seventy four percent of the 
teachers In both groups said they follow their teacher^s guides. 

Both! 'groups agreed that over 80 percent of theTr "students_should be 
able to master any skill In their reading program. They perceived less 
than 20 percent of their lessons to focus on Main Idea, and that almost a 
third of their students (32%) knew Main Idea before the lessons, and only 
60 percent of their students would master Main Idea. The observed teachers 
an^ questionnaire respondents were also In close agreement about tlve 
problems their students have with Main Idea. Thirty percent of the 
questionnaire respondents and 36 percent of the observed teachers said that 
their students confuse main Idea with the titles of stories. A little over 
40 percent of each group^s students (42% and 45%) think the Main Idea is 
the first sentence of a story, and almost half (46% and 54%) think the Main 
lidea must be a sentence pulled directly from the text. Similar percentages 
(42% jand_48^ be unable to pick out _the cj)rrect 

main idea if it is not expressed in a sentence in the text. A little over 
a third (36% and 44%) expected their students would not to be able to 
generate a Main Idea sentence. Table 6 summarizes the ways the observed 
teachers and questionnaire respondents were similar. 



Insert Table 6 about here. 



In addition to these areas in which the observed teachers and the 
questionnaire respondents were very similar in describing their own 



teaching and management, there were also a few areas 



in which the two 



groups differed. The questionnaire respondents claim that their students 
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miss almost twice as many (18% to' 11%) instructional items in their reading 
lessons as the observed teachers* But, the observed teachers claim to 
correct twenty percent more student errors (88% to 68%) than the 
questionnaire respondents* Both groups of teachers say they ignore 
mistakes infrequently — 15%-25% of the time. The teachers also differ on 
the frequency with which they tell students the answer and then repeat the 
quesHott elthex.^^^ individual or to the group to correct mistakes # The 
observed teachers practicie both of these "fypesr*^^ — 
twice as often (47% to 19% and 57% to 22%) as the questionnaire teachers* 

0 

Likewise, the teachers disagree about when it is appropriate to help or 
gjiide students, except that they agree that it is appropriate to help them 
only 25% and 35% of the time if questions are open-ended* The two groups 
of teachers differ with respect to helping students with the observed 
teachers feeling it is appropriate to help students more often than the 
teachers who responded to the questionnaires* The areas of disagreement 
were if: (1) a skill is new (100% to 79%), (2) students won* t master the 

y skiil-iai^-hout- help- (88%^ af ^^^^t the -t ime) , and ( 3 ) students 

have not yet been taught the skill (88% of the time to 54% of the time). 

Part 3* Student Performance and Teacher Expectations 
at the Conclusion of the Main Idea Lesson 
This section of this paper compares how the observed teachers expected 
their students to perform on criterion-referenced items designed to measure 
student performance on a range of reading comprehension areas, with* the 
students* actual performance. 
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^ Method 

An additional area investigated in this work was the relationship 
between student performance on objective items designed to measure 
information presented in the two lessons videotaped for the observed 
teachers and the performance the teachers predicted their students would / 
obtain* Thel test included items from the following eight categories: Main 
Idea, Key Words, Map Skills, Inference, Context Clues, Relevant Details, 



Cause/Ef fee 



, and Fact /opinion* The purpose of this measurement was to see 



how accurately the teachers evaluated their students* performance after the 
Main Idea ijesson* 

\ These data were collected immediately after the teachers finished 
teaching the lessons to capture student performance immediately after the 
lesson* As students took the criterion-referenced tests, teachers 
responded tq the interviewer^s questions* 



\ Results 
Table 7 ^s hows the results from the interviews and tests* To the 
question, "Whi^t percent of the students should master any skill?" the 
teachers responded 86%, though only 12% of their students then performed at 
or above 90% oii all topics* Thirty percent obtained mastery if the 
criterion of performance Ir lowerea to 75%, and 55% performed at the 50% 
criterion* 

The teachers thought that over half their students had mastered the . 
main idea items, though only 10% of the students actually performed at or 
above the 90% criterion, 33% performed at or above the 75% criterion, and 
58% above the 50% criterion on Main Idea* 
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Insert Table 7 about here* 



Table 8 sho<i^s the percentage of students who scored at 90, 75, and 50% 
on each topic. Of these eight categories, by far the highest student 
performance Is on Relevant Details. The "relevant details" Items are what 
are traditionally thought of as literal, or text explicit, comprehension 
questions* Twenty-four percent, eighty-two percent, and ninety-nine 
percent of the students are at the 90% correct, 73% correct, and 50% 
correct levels, respectively. The overall lowest student performance Is on 
the Context Clues Items with only 15% of the students scoring at 50% « 
Similar percentap:es of students perform at the 50% level (58%, 56%, and 
60%) for Main Idea, Map Skills, and Caui^e/Ef feet . There Is similar 
performance for Key Words and Fact /Opinion with 65% and 70% of the students 
testing at or above the 50%# The average performance across the eight - 
:ategorles has 12% of the students answering 90% or more of the Items'.'' 
correct, 30% of the students at or above 75% correct, and only a little 
more thau half (55%) at or above 50% correct. 



Insert Table 8 about here* 

... — ^ 

In summary, there were several types of data presented in these 

studies: (1) an analysis of parts of four basal reading textbooks used in 

« 

fourth, fifth, and sixth grades; (2) observations of general teaching 
behaviors for teachers teaching two comprehension lessons; (3) a report of 
teachers* expectations for student performance after the taped lessons from 
interviews and questionnaires; and (4) student performance on criterion- 
referenced tests designed to measure the skills taught in thelt reading 

15 



lessons. The next section of this paper provides a discussion to Integrate 
these data and presents the Implications of these findings. 

Discussion 

basal textbook analyses . The analyses of Mala Idea Instruction In 
four basal reading textbooks (for grades four^ five,. and six) showed an 
average of 22 lessons for the three year period — about 7 lessons per year. 
The sparse number of lessons coupled with jio specified correction 
procedures for students mistakes suggests that there may be less than 
adequate Instruction and practice on this reading comprehension skill for 
most students. . 

In addition^ a high percentage of the presentations are ambiguous, 
with many misleading or Incorrect answers. The ambiguity of the questions 
and the wrong answers given complicate the instruction substantially. For 
example in one fourth grade program, an early lesson in the workbook de^lt 
with the "plot" of a story. A lesson about halfway through the book 
introJuced understanding the "topic" of a paragraph, while a lesson just 
one story later introduced Main Idea and details. Many of the questions in 
the workbook could easily be int;,erchanged to focus on either Main Idea, 
plot, or topic, thus posing a confusing situation for teachers and 
students. 

Variations in wording and format from example to example may present 

additional problems. While some authors might sec variations in wording in 

t 

either tihe students workbook or the teacher^s presentation as a positive 
feature of their matei^^als, it is doubtful that most variations are either 
helpful or necessary for lower-performing students. And, given the meager 
number of Main Idea lessons that students are exposed to, it la doubtful 
that they will either tire of or, become dependent upon one pattern of 
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wording. While some might take issue with the way the number of 
lessons/level were calculated to determine which activities constituted 
each "lessoQ^" the lessons presenting Main Idea are so few and far between 
that even very different metric that yielded a few dayg more or Ijpss 
between the activities could not make much difference. 

These results are similar to those found by Beck and McCanlin (1978) 
and Beck et al. (1979) as they searched first (1978) for components in the 
first two levels of basals that would present adequate code-breaking 
skills, and then searched and analyzed again (1979) to see what kind of 
exercises are in the basals for teaching reading comprehension. They found 
a limited number of lessons devoted to skill instruction and practice. 
These findings also support Durkin (1981) when she concluded, "When 
instruction does appear in manuals, the connection between what is being 
taught and how to read is either minimized or entirely overlooked" (p. 
542). Durkin continues, "As a result, identifying referents for pronouns, 
reading with *a big voice,* distinguishing between facts and opinions, 
finding topic sentences — all these activities become ends in themselves" 
(p. 542). Durkin was describing basal manuals for kindergarten through 
sixth grade, and her findings are replicated in the first portion of this 
work. The lack of direct teaching and adequate practice would probably 
make the reading task difficult for most students. 

Teacher observations * The second study looked at general teaching 
behavior — such as those identified as generic direct teaching behaviors by 
Stevens and Rosenshine (1981) and unique to the University of Oregon Direct 
Instruction Model (see, for example, Meyer, Gersten, & Gutkin, 1983). The 
overall slow presentation rate (4.2 responses/minute — one response roughly 
every 15 seconds) suggests a great deal of teacher talk with student 



17 

19 



responses sprinkled In just occaslonally# The high percentage of questions 
to individual students, low percentage of errors corrected (37%), and the 
high ratio of general, Instead of behavior-specific praise suggests an 
overall slow-moving teaching sequence focused primarily upon individual 
students, most probably the high performing students. These teaching 
behaviors are not surprising given the information gleaned from the 
analysis rf the basal textbooks. Recall, for example, that there are no 
correction procedures specified in the texts. 

When the teaching behaviors were compared to the program 
specifications, the biggest difference in program specification and 
teaching behavior was that the teachers averaged 20 questions per lesson 
while teaching Main Idea, though the program specified only 8 questions on 
the average. An analysis of the video tapes provided a simple explanation 

■J 

for this. 251% difference between. the number of questions that the programs 
specified and the number of questions the teachers asked. Perhaps because 
the programs lacked correction procedures, teachers often corrected 
stude^its by asking additional questions. Therefore, the additional 
questions are in" part an~^ index" of i:he number of mistakes t^ie-^i^dents made.- 

This discrepancy in the number of questions asked may also explain the 
difference in the percentage of the program questions relevant to the topic 
(69%) compared to the percentage of teacher questions relevant to^the topic 
(24%). Teachers often used additional queptA.jns as correction procedures. 
Often, these "correction questions" were irrelevant to the topic. The 
other areas examined comparing teacher behavior and program specifications 
suggest that the teachers model their behavior after thefr programs guides. 
(This was also confirmed by interviews.) 

Interview and questionnaire responses . The questionnaire responses 
showed very few areas (about 12%) where the 8 teachers observed differed 
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from the 493 representative teachers who returned the questionnaire. The 
two groups of teachers gave precisely the same answers when asked If they 
followed their basal programs — three fourths of the teachers in both groups 
said that they followed their teaching materials. They did, however, 
misjudge the content of tLeir programs. Similar data have recently been 
gathered by Mason (Mason, note 4) when she h<ad teachers search for examples 
of various reading comprehension activities in^t»elence and social studies 
materials. Mason reports that teachers were often surprised when they 
could not "find" lessons on a topic such as main Idea, or cause/effect. As 
noted previously, the teachers also drastically overestimated the 
percentage of their students who had mastered main idea and othet skills. 

c 

I 

tested on the criterion-referenced tests developed for this study. 

Student outcomes. Overall student performance on the eight categories 
tested on the criterion-referenced tests was low — only 12% of the students 
reached the 90% criterion^^ 30%"The 75% critlBrion, and 55% the 50% 
criterion. This overall low performance demonstrates that very few 
students mastered any of the "higher order" comprehension skills such as 
main idea or cause/effect. 

These scores are particularly disturbing in light of the teachers* 
expectations. The teachers expected 86% of their students to master 
any /all of the reading comprehension skills. The teachers thought that 
over half of their students (56%) had mastered Main Idea when in effect 
only 10% of the students tested were at or above 90^ mastery on Main Idea. 

These crlterion--ref erenced test scores may be cause for alarm for two 
reasons. Fltst, these scores demonstrate how poorly students are 
performing on these skills. Students have failed to master virtually all 
of the skills. Second y and perhaps even more alarming, teachers expected 
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their students* performance to be much higher than it was* T achers 
perceived that most of their students had mastered these reading 
comprehension skills and could perform the related workbook tasks. In fact 
the teachers' judgment was far from accurate. This discrepancy suggests 
that teachers either misinterpret pheir students' performance or evaluate 
incorrectly aspects of their students ' performance. Another explanation 
for student performance that is so far below their teachers' expectations 
is that teachers simply had an inadequate number of criterion measures in 
the basal lessons upon which to make accurate judgment. In addition, if 
most of their individual turns went to their higher perfo^fmers, it is easy 
to see why teachers might have the percept iojis_tjiey .Jtr^. 

. /. 
Implications 

y 

The results of this study hav^ far-reaching implications. First, the 
analysis of the texts revealed that\ there are actually few lessons in the 
materials that "teach" Main Idea. 'From the overall student performance on 
the criterion-referenced tests, it is probably safe to wager that basals 
cover other reading comprehension skills as poorly. One message to the 
dLiVelopers of basal readers is clear — there simply is not enough 
systematic, .well-designed instruction and practice on reading comprehension 
skills in the present materials. 

It is therefore no wonder that compensatory education students 
continue to score poorly on norm-referenced and criterion-referenced tests 
of reading comprehension. The basal reading programs must be revised to 
include more direct teaching and practice in reading comprehension. 
Students would no doubt benefit from practice on expository as well as 
narrative text. At this time, most reading instruction in the middle 
grades continues on narrative texts, though students are expected to handle 
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expository material and learn to read in the content areas, Wouldn*t it 
make sense to provide reading practicj^ on expository texts during in the 
basal reader? 

The average performance of 'these teachers is also alarming. By 
presenting items slowly to individual studepts, and by correcting just over 
a third of the errors, there is room for a great deal of improvement in 
their overall teaching performance. 

The results of this study also suggest that teachers need to be taught 
how to evaluate student performance. The frequent discrepancy between what 
teachers perceive and report and what is observed was documented earlier by 
Jlook and Rosenshine (1979), The obvious conclusion based upon the 
discrepancies._between the teachers' self-report data and observational data 
is that teachers perceivfv and/or report their behavior and their students* 
behavior inaccurately. How might we -improve this situation? If teachers 

r ^ 

gave frequent, valid criterion-^ref erenced tests and/or some other type of 
monitoring procedure to secure objective, accurate assessments of their 
sutdents they might judge the students' performance more accurately. If 
students were tested often on what they had been taught, and if teachers 
interpreted the test results quickly, they could remediate problems 
efficiently before continuing iastruction and thereby continuing to build 
on a shaky foundation. 

There is substantial evidence that a student monitoring system such as 
the one suggested above is one of several variables correlated to increased 
student achievement in effective schools for compensatory education 
students (Edmonds, 1979), Criterion-reiferenced monitoring has also been 
identified as an important variable in a variety of schools that have 
adopted the Direct Instruction Follow Through Model (see Meyer, Gersten, & 
Gutkin, 1983; or Meyer, 1983 for descriptions of this criterion-referenced 



monitoring system). In fact, such a testing system usually help^ teachers 
become objective in their assessment of their students simply by providing 
frequent feedback to teachers on all students. 

Extensive experience with a criterion-referenced testing system 
developed for the Direct Instruction Follow Through model has led one 
district to develop a similar testing system for their basal readers 
(Coste^lOy Note 5). Teachers administer these tests every six weeks to 
determine student achievement in their basal readejrs just as they do in the 
Direct Instruction materials. 

Whi;^j^e the short-term objective would be to develop paper and pencil 
procedures for monitoring student achievement, a long-term goal would be 
for teachers to become more sensitive to the performance of their students, 
and more direct in their teaching so that they would be constantly 
'^testing*' as they teach all students to be certain that students have 
mastered skills. 

0 




22 



24 



V 



Reference Notes 

1. Anderson, R. C, Mason, J. M., & Sbirey, L. The reading group: An 
expferj-mental -investigation of a labyrinth (Tech. Rep. No. 271). 
Urbana; University of Illinois, Center for the Study of Reading, 1983. 

2. Carnlne, L. K., Carnlne, D. W., & Gersten, R. M. Analysis of oral 
reading errors mad, by economical l,y disadvantaged students taught with 
a. synthetic p honics approach . Unpublished manuscript. Eugene; 
Unlverf '.ty of Oregon, 1983. 

3. Osborn, J. The purposes, uses, and contents of workbooks and some 
guidelines for teachers and publishers ^Reading Education Rep. No. 27). 
Urbana: University of Illinois, Center for the Study of Reading, 1981. 

4. Mason, J. M. Personal correspondence, 1983. 

5. Costello, A. Personal correspondence, 1983. 



ERIC 



23 



25 



References 

Aaron, E. E.,. Jackson, D., Riggs, C, Smith, R. G*, Tierney, R. J. Scott> 
Foresman Reading s Glenview, IL: Scott, Foresman and Cciapany, 1981. 

Beck, I* L. Instructional Ingredients for the development of beginning 
reading competence . Paper prepared for the National Conference on 
Urban Eduqatlon, CEMERL, Inc., 1978. 

Beck, I. L. , & McCaslln, E. S. An analysis of dimensions that affect the 
development of codg-breaklng ability in eight beginning reading 
programs . University of Pittsburgh, Learning Research and Development 
Centet, 1978. 

Beck, "I. L,, McKeown, M. G. , McCaslin, E. S. , & Burkes, A. M. 

Instructional dimensions that may affect reading comprehension; 

Examples from two commercial reading proferams . University of 

Pittsburgh, Learning Research and Development Center, 1979. 
Clymer, T. , Christenson, B., & Brovm, V. Reading 720 . Lexington, MA: 

Ginn, 1977. . _ 

Durkin, D. What classroom observations reveal about reading, comprehension 

instruction. Reading Research Quarterly , 1978-79, }M^) , 481-533. 
^Durkin, D. Reading comprehension ^instruction in five basal reader series. 

Reading Research Quarterly . 1981, j^(A), 515-544. 
Durr, W. K. , Lei »re* J. M., & Alsin, M. L. The Houghton Mifflin Reading 

Series ^ Boston, MA: -sHgughton Mifflin, 1976. / 
Edmonds, R. Effective sch^ls for the urban poor. Educational Leadership , 

1979, 16, 1-16. 



24 

26 



Engelmann, St E. , & Bruner, E. C. Dlstar Reading I & II, Chicago; 
Science Research Associates, 1974, 1975. 

Evertts, Weiss, Crulkshank, Steuer, Rosenbaum, Shaw, & Tolbert. Holt 

Basic Reading Program * New York: Holt, Rlnehart, & Winston^ 1977. 
Gall, M. D. Handbook for evaluating and selecting curriculum materials , 

B'-ston: Allyn and Bacon, 1981. 
Gilm, R, E. The Palo Alto reading pro^^ram (2nd ed. ); Sequential steps In 

reading . New York: Harcourt Brace Jovanovich, 1973. 
Hook, C. M., & Rosenshlne, B. V. Accuracy of teacher reports of their 

classroom behavior. Review o f jlducatlonal Research , 1979, 49(1), I-- 

12. ^-"^ 

Meyer, L. A. Long-term academic effects of the direct instruction project 
follow through. The Elementary School Journal , in press. 

Meyer, L. A., Gersten, R. M., & Gutkin, J. Direct instruction: A project 
follow through success story* Elementary School Journa l, 1983, 84(2), 
241-252. ^ 

S t everis, R. , & Roserishtrie , B V Advances in ^research on teaching. 
ExcepirtQjial Education Quarterly , 1981, 2^, 1-10. 




Table 1 

Frequency and Clarity of Main Idea Presentations in the 
Four Basal Readers Grades 4> 5, and 6 



Mean Across 
Programs 



Frequency 

1. Total number of program examples of main idea 66 

2/ Number of student e:R;amples on the saue day as teacher 

material 9 

3. Total number of lessons in which main idea is presented 22 



4. Average lesson days since two examples were presented 



62 



Clarity 

1. Percentage questions ambiguous and not taught 88 

2. Percentage of questions misleading or wrong answer 

-iS'^givfen-""^ ■ - • ■ - ..... ... — ..12 - 

3. Probability of a correct interpretation. (Implies 
an average of 4 interpretations possible per lesson 
sequence) * 27 

4. Percentage of variation in: • 

a. ..student workbook wording 44 

b» studeiit-^orkbook items forms 18 

c.V teacher presentation wording 14 

teacher Items-forms - - V- ■ _ . _:_ 10 

5. Percentage student workbook: 

a. response variation . ^ 13 

b. visual distraction 25 

6. Percentage of examples for which a correction is 

specified / . 0 
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Table 2 

Teachers' Reading Comprehension 
Lessons Video-Taped 



Teacher 




Topic A 


Topic B 


1 




Story types 


Main idea 


2 




Main idea 


Map skills 


3 




Story comprehension 


Main idea 


4 




Main idea 


Story comprehension 


5 




Vocabulary 


Context clues- 


6 




Story comprehension 


Main idea 


7 




^ Parts of a book 


Parts of a book 


8 




Parts of a book 


Main idea 


9 




Main idea 


Map skills 


10 




Map skills ' . 


Main idea 


11 




Communications lab. 


Main idea 


12 




— Vocabulary 




13 




Vocabulary 


Keywords 


14 




Dictionary skills 


Inferences 


15 




Verbs 


Context clues 


16 




Hataphor /simile 


Main idea 


17 




Main idea 


Cross references 
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Table 3 
Observed Teaching Behaviors 



Activity Percentage 



Teachers* Presentacional^Behaviors 

Group Tasks 16% 

Individual Tasks ' 84% 
Student Responses/Minute 4.4% 

Student Error-Rate to Teacher Questions and Corrections 

Student Errors 27% 

Error Corrected by Teacher 37% 

Errors Corrected and Retested by Teacher 10% 

Teaching Strategies 

Models 20% 

Leads 1A% 
Test's '■ 66% 

Teacher Feedback 

General Praise 44% 

Specific Praise 2% 
Negative Feedback ^ 1% 

No Feedback 53% 
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-Table 4 

Comparison of Teacher and Program Questions 



Question Aspect Programs (N » 4) Teachers (N « 17) 

Average number of questions 

asked per lesson 8 '20 



Percent of questions ambiguous 

or mislead ng 42% 



Percent relevancy of questions 

to reading topic 69% 24% 



Percent correct student responses 78% 77% 

Percent prompted responses 12% 22% 



. Table 5 

Teachers' Anticipated Student Outcomes 
and Program Characteristics 
Teacher N = 17 



Percentage of students that: 

Should master all skills 86% 

Should do workbooks accurately after lesion 72% 

" Need more practice on topic taught 58% 

(Amount of additional practice needed — 1 week) 

Should master main idea 56% 
How deficient is the program for teaching 



main idea? 



16% 
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Table 6 

Comparison of Similarities Between 
Observed Teachers and Questionnaire Respondents 



Category 



Observed Teachers 
(N = 8) 



Questionnaire 
Respondents 
(N = 493) 



Number of students taught 
Description of class 
Organization for reading 
Hours per week teaching reading 



28 
avejrage 
small groups 
6.2 



26 
average 
small groups 
5.5 



Percentage of groups reported: 

to Follow Teacher's guide 

Percent S's should master any skill 

Percent lessons focused, on main idea 

Percent S's who knew main idea before 
instruction 

..PercanL who. WQuld.ma.ster main idea. 



74% 
84% 



32% 
.60%. 



.60%. 



Teachers* perceptions of Student Problems 

Percent who confuse main idea with title 

Percent who think main idea is the first 
sentence 

Percent who think main idea must be a 
sentence from text 

Percent who can't pick out main idea 

Percent who can't generate main idea 
sentence 



45% 

54% 
48% 

44% 



30% 



46% 
42% 

36% 
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Table 7 

Teacher Expectations and Student Performance 



1 


a. 


What percent of the students should master 
any reading skill? 


86% 




b. 


Percent at 90% cut all topics 


12% 




c* 


Percent at 75% cut all topics 


30% 




d- 


Percent at 50% cut all topics 


55% 


2 


a. 


What percent of students should master main idea? 


56% 




b. 


Percent at 90% cut main idea 


10% 




c. 


Percent at 75% cut main idea 


33% 




d. 


Percent at 50%" cut main idea 


58% 
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Table 8 

Student Criterion-Referenced Scores in Eight Reading 
Comprehension Categories 



Comprehension 
Skill 


Percent S's at 
or above 90% 


Percent S's at 
or above 75% 


Percent S's at 
or above 50% 


Main Idea 


10 


33 


58 


Key Words 


8. 


32 


65 


Map'' ^ills 


30 


33 


56 


Inference 


15 


. 30 


62 


Context Clues 


0 


0 


15 


Relevant Details 


24 


82 


99 


Cause/Effect 


10 


• 30 


60 


Fact/Opinion 


0 


25 


70 
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